The present invention relates generally to a distributed computation method, and more particularly, but not by way of limitation, to a system, method, and computer product used for a node of a distributed computation system.
With fast development of computation technology, applications have emerged in which need huge computation capacity. Distributed computation technology has been known in which an application is divided into a number of small parts, and the small parts are distributed to a plurality of computing devices for parallel processing. By virtue of the distributed computation technology, the overall time for the computation can be reduced and the computation efficiency can be improved.
A so-called “Hadoop” system has been developed as an advanced distributed computation framework which allows for execution of large applications on large groups of inexpensive hardware devices. A Hadoop system has a distributed file system called HDFS (Hadoop Distributed File System) in which a file may be divided into one or more blocks and the one or more blocks may be stored in a plurality of computing devices. In addition, in a Hadoop system, operations in an application are abstracted into Map operations and Reduce Operations. A Map operation is to process an input Key/Value pair and to output a Key/Value pair, and a plurality of Map operations may be performed in parallel. A Reduce operation is to work on the plurality of Key/Value pairs output from the Map operations and to output one or more Key/Value pairs the number of which is typically smaller than that of the plurality of Key/Value pairs.
Resource allocation in version 1.0 of Hadoop is in unit of “slot” which is a minimum resource allocation unit, and the resource in each slot is static and unified. Resource allocation in version 2.0 (also called “Yarn”) of Hadoop is in unit of “container”, and the resource in each container is user defined but static.